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DESCRIPTION 
INFORMATION TERMINAL 

Technical Field 
5 The present invention relates to an information 

information terminal, and specifically to an information 
terminal including an interpolation device. 

Background Art 
Recently, information terminals equipped with cameras 
10 have become popular, and have also been employed as means 
for imaging, browsing and image communication. 

For example, in the case of a cellular phone equipped 
with a camera, an image acquired by the camera is used in 
various ways, such as being transmitted as a file attached 
15 to email text, being used for real-time communication (a 
television telephone), or being printed by a user. The 
same uses may be made of an image received from a third 
party and of an image downloaded from a server. 

Therefore, originally, in order to increase the 
20 usability, it is preferable that the image quality or the 
speed required for the processing, i.e., the amount of 
processing for an operation or the amount of processing 
memory, be changed in accordance with the use of a 
photographic image or a received image, as described above. 
25 For example, as shown in Fig. 5, "processing speed has 

priority over image quality" when a photographic image is 
to be used for real-time communication. On the other hand, 
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for example, when a photographic image is to be printed out, 
a case wherein "image quality takes priority over 
processing speed" is more meritorious for a user. 

Disclosure of the Invention 
5 However, generally, a conventional cellular phone 

merely provides a fixed image quality and a fixed 
processing speed, regardless of what is favored by a user 
and its use. 

Furthermore, since downsizing and a reduction in power 

10 consumption and its costs are indispensable for an 

information terminal, such as a cellular phone, it is not 
easy to mount a camera, such as described above, that 
performs flexible image processing. 

Further, since downsizing, a reduction in power 

15 consumption and costs are also indispensable for the 

processor of an information terminal, it is not generally 
projected that all of the image processing ordinarily 
performed by a camera be shifted to the processor in order 
to guarantee the flexibility of the camera image processing. 

20 While taking the above points into account, one 

objective of the present invention is to provide an 
information terminal that can minimize alterations of a 
camera and a processor of the information terminal , and 
that can flexibly change image quality and processing 

25 speeds, at a low cost and within a short period of time, in 
accordance with the usage states and other, varied 
conditions, such as the extent of operations, required of 
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the imaging processing performed by the camera and the 
information terminal , and the amount of processing memory 
required for the operation. 

It is another objective of the present invention to 
5 provide an information terminal that can reduce the 

influences due to the simplified processing, performed by a 
camera DSP, and that can qualitatively increase the value 
of an image. 

It is an additional objective of the present invention 
10 to provide an information terminal having a function that, 
in accordance, for example, with a use state, flexibly 
increases image quality one level and exceeds that of an 
information terminal equipped with a camera having the same 
function . 

15 It is a further objective of the present invention to 

provide an information terminal that can improve the image 
quality, either without requiring a change in the hardware 
architecture of a camera module mounted on the information 
terminal or without increasing the size of a camera DSP. 

20 To achieve these objectives, according to a first 

invention of the present invention, an information terminal 
is characterized by comprising: 

data operation processing means, for performing 
operation processing for input image data and preparing 

25 output image data; 

removal means, for removing, from the output image 
data, part or all of the steps of an operational processing 
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sequence performed for the input image data; and 
data processing means , for performing other 
operational processing steps for data obtained by the 
removal means and for preparing output image data. 
5 The image data input to the information terminal are, 

specifically, color image data based on a Bayer-type RGB 
arrangement, i.e., an arrangement of red, blue and green 
pixels that generally is used by a single-chip image sensor 
of a color camera. 

10 The data processing means for the information terminal 

is specific means for performing interpolation processing, 
and more specifically, is means, when a sensor outputs R, G 
or B pixels, for employing the values of adjacent pixels 
having the same color to perform interpolation processing 

15 using the nearest neighbor method, the linear approximation 
method or the cubic convolution method for pixels G and B, 
R and B, or R and G that have been eliminated by a color 
filter that is laid on the surface of the sensor. 

It is preferable, when irreversible processing has 

20 been is performed for the image data input to the 

information terminal and the output image data have been 
prepared, that the removal means perform inverse function 
transformation processing to remove part or all of the 
interpolation processing performed for the input data. 

25 It is preferable that the data processing means of the 

information terminal again perform an edge enhancement 
process. 
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Furthermore, according to a second invention of the 
present invention, an information terminal is characterized 
by comprising: 

a camera module, which has an image sensor for 
5 digitizing and using a signal output by the image sensor to 
prepare first Bayer-type data, and for employing the first 
Bayer-type data to prepare first image data, using a first 
algorithm, and transmitting the first image data; and 

a host module, having a main storage device, for 
10 receiving the first image data obtained by the camera 

module and storing the received first image data in the 
main storage device, for reading the first image data from 
the main storage device and extracting second Bayer-type 
data from the first image data that have been read, and for 
15 employing the second Bayer-type data to prepare second 
image data using a second algorithm. 

It is preferable that the first image data and/or the 
second image data have an RGB form or a YUV form. 

It is preferable that the first image data provided be 
20 compressed. 

It is preferable that the host module include a data 
output unit for outputting data to a printer, so that the 
second image data are output to the printer through the 
data output unit. 
2 5 Brief Description of the Drawings 

Fig. 1 is a functional block diagram showing the 
periphery of a retrial module according to one mode of the 
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present invention; 

Fig. 2 is a functional block diagram showing the 
periphery of the retrial module according to the mode of 
the present invention; 
5 Fig. 3 is a functional block diagram showing the 

periphery of the retrial module according to the mode of 
the present invention; 

Fig. 4 is a flowchart for the retrial module according 
to the mode of the present invention; 
10 Fig. 5 is a diagram showing a relationship between the 

purposes of the use of an information terminal and 
requested items ; 

Fig. 6 is a diagram for explaining a first method for 
the interpolation processing; 
15 Fig. 7 is a diagram for explaining a second method for 

the interpolation processing; 

Fig. 8 is a diagram for explaining a third method for 
the interpolation processing; 

Fig. 9 is a diagram showing the external appearance of 
20 a cellular phone equipped with a camera; 

Fig. 10 is a diagram showing the hardware 
configuration of the cellular phone equipped with the 
camera; 

Fig. 11 is a diagram for explaining the creation and 
25 re-creation of image data; and 

Fig. 12 is a flowchart for a photographing mode. 
Best Modes for Carrying Out the Invention 
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1. First Mode 

A first mode of the present invention will now be 
described while referring to the accompanying drawings. 
(1) Explanation for encoding photographed images 
5 Fig. 1 is a functional block diagram for explaining 

this mode of the invention, and for showing only modules of 
an information terminal that are required to explain the 
present invention. In this mode, a photographed image is 
encoded into a predetermined form (a JPEG form in Fig. 1) 
10 to be stored or to be transmitted, for example. 

A camera module 103 includes a sensor and a camera DSP. 
The sensor has, for example, a color filter to cope with 
the three primary colors of light, blue (B) , green (G) and 
red (R) , and in this filter, G is arranged in 
15 checkerboardwise and R and B are sequentially arranged 
linearly. In this specification, this arrangement is 
called a Bayer-type arrangement. Image data that are 
output by the camera module that employs the color filter 
having the Bayer-type arrangement also have a Bayer-type 
20 arrangement. 

When a user points a lens 101 at a subject and 
releases the shutter, the sensor generates image data that 
include a Bayer-type arrangement consonant with the 
detected subject, and the camera DSP performs operations 
25 that are image quality correction means that includes gamma 
correction means, color interpolation means and an edge 
enhancement process and other processes. 
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During these operations, the individual color elements 
of the image data having the Bayer-type arrangement, i.e., 
the R element, the G element and the B element , are 
extracted; the color correction means adjusts the balance 
5 of the color elements; the gamma correction means 

transforms the values of the pixels for the individual 
color elements independently, by using transformation 
expressions or a lookup table; and the color interpolation 
means performs interpolation processing, so that missing 

10 color elements, for the pixel positions of the individual 
color elements, are provided, for adjacent pixels having 
the same color elements. As a result, the color elements 
that are removed by the color filter laid on the image 
sensor are restored for all the pixels. 

15 In this case, since minimization of the chip area and 

reductions in size and cost are requested, problems could 
occur in the operating ability of the camera DSP; for 
example, a complicated operation may be disabled that 
requires many operations or a large capacity processing 

20 memory, such as a line memory, to perform image processing, 
an extended period of time may be required even when such a 
complicated operation is enabled, or synchronization with a 
succeeding processing step of receiving processing results 
may not be obtained. 

25 Therefore, the color correction means, the gamma 

correction means and the color interpolation means 
available at this stage are those for which the operational 
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processing and the processing memory size are reduced as 
much as possible. In particular, for the color 
interpolation means, which is one of the image processing 
means that most requires memory and operational processing 
5 resources, an example can be provided that, relative to 
adjacent pixels, for example, employs a linear 
interpolation method. 

Similarly, for the edge enhancement processing that, 
comparatively, consumes as many processing resources, there 

10 are many examples wherein, in order to optimize the amount 
of processing memory and the amount of operational 
processing, edge enhancement filters having horizontally 
elongated and vertically shortened taps are employed that 
can perform the processing but that use a smaller number of 

15 line memories. 

Several other image quality correction processes are 
performed for the image data obtained by the above 
described various processing means, and after the 
processing sequence for the camera is completed and when 

20 the storage of image data is the purpose, compression 

processing, such as JPEG compression, is performed for the 
image data, and the compressed image data (hereinafter 
referred to as processed data) are output to storage means 
111. When the display of image data is the purpose, image 

25 quality correction processing is performed for the image 
data to obtain the optimally limited number of pixels and 
number of colors for display means 113, and the thus 
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obtained image data are output to the display means 113. 

A retrial module 107 includes removal means 107a, 
color interpolation means 107b and image quality correction 
means 107c, and data output by the camera module, i.e., 
5 processed data, are transmitted to the removal means 107a. 

When the retrial module 107 is started, the removal 
means 107a removes part or all of the image processing that 
has been performed for the processed data by the camera DSP. 
The processes removed here are, for example, the color 
10 interpolation process, performed for image data having a 

Bayer-type arrangement, and/or the edge enhancement process, 
which is a typical step performed as part of another image 
quality improvement process. 

When the interpolation process has been removed from 
15 the interpolated data, data having a Bayer-type arrangement 
are recreated. In this case, the removal of the 
interpolation process consists of the performance of an 
intermittent process by which the R, G and B elements of a 
color image, obtained by the series of camera processing 
20 events, are arranged in consonance with the R, G and B 
Bayer-type arrangement for the image sensor. 

When the color elements of the individual pixels 
obtained through the performance of the color interpolation 
process are reset, restrictions imposed on the color 
25 interpolation processing and the image quality improvement 
processing that are provided by the camera DSP are reduced 
to their lowest levels. 
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Thereafter, the data obtained in the removal process 
are transmitted to the image quality correction means 107c, 
the retrial module 107 performs an interpolation process 
that differs from that performed by the camera DSP and, if 
5 needed, an edge enhancement process that differs from that 
included in the image quality improvement processing for 
the camera DSP, and the resultant data are transmitted to 
JPEG encoding means 109. 

An interpolation process that requires a more 

10 complicated operation, an increased amount of operational 
processing and a greater amount of processing line memory 
can be employed as another interpolation method. Similarly, 
a process that requires a more complicated operation, an 
increased amount of operational processing and a greater 

15 amount of processing memory can be employed as an edge 

enhancement process that is included in other image quality 
improvement processing . 

According to this mode, for example, an interpolation 
process that uses the nearest neighbor method is provided 

20 by the camera DSP, and is performed by employing an amount 
of processing memory that is the equivalent of two lines of 
image data, without any multiplication being required. 
Then, the retrial module 107 employs an interpolation 
process, using a cubic convolution method, to perform 

25 floating-point multiplication for a quantity of processing 
memory that is the equivalent of five lines of image data. 
Furthermore, the interpolation process performed by 
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the camera DSP includes the image quality improvement 
process that uses an edge enhancement filter wherein taps 
for five horizontally arranged pixels are provided for each 
line of an image. The retrial module 107 provides the 
image quality improving process, using edge enhancement 
filters, in which taps for five horizontal pixels are 
provided for each three line group in an image, that are 
changed in accordance with the set of values in each block 
of 3 x 5 pixels. 

Of course, the present invention is not limited to 
these specific color interpolation and edge enhancement 
processes, and has a general processing architecture. 

According to this mode, only when the retrial of the 
interpolation process is required is the retrial module 107 
started, and when the retrial is not required, processed 
data are transmitted directly from the camera module 103 to 
the JPEG encoding means 109. 

Whether or not the retrial is required is visually 
determined when an image is photographed and is displayed 
on the screen provided by the information terminal. 

The start of the retrial module 107 is instructed, by 
the manipulation of a user menu or a key button prepared 
for the information terminal, when it is visually 
determined that the image quality is unsatisfactory, such 
as when an image is slightly blurred or there are jaggies 
along the edges. 

Of course, processing means for automatically 
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determining whether an image quality is satisfactory , and 
processing means that employ the results obtained by 
automatic determination to automatically start the retrial 
module 107 may be provided. 
5 The retrial module 107 may employ processing that 

differs from the processing provided by the camera DSP, in 
addition to such image quality improvement processing as 
the interpolation process and the edge enhancement process, 
or may employ different processes as only parts of these 

10 processes. 

In the mode described above, before the JPEG 
compression process is performed, immediately after an 
image is photographed, whether the retrial module 107 
should be started is determined and the processing included 

15 for the retrial module is performed. However, as is shown 
in Fig. 3, these process sequences may be performed for an 
image obtained using JPEG compression. 

That is, JPEG decoding means 305 performs the JPEG 
decoding processing for JPEG compressed image data and 

20 displays the resultant data on a screen, and a retrial 
module 307 is started and the processing results are 
transmitted as processed data to JPEG encoding means 309. 
(2) Explanation for outputting of decoded image 
Fig. 2 is a functional block diagram for explaining 

25 another mode according to the present invention, and shows 
only modules that constitute an information terminal 
according to the present invention and that are required to 
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explain this invention. According to this mode, an image 
obtained as a predetermined form (a JPEG form in Fig. 2) is 
output and stored. 

An explanation will be given for an example wherein 
5 JPEG decoding means 205 decodes a JPEG image, again 

performs the processing for the obtained image, and outputs 
the results. 

A retrial module 207 includes removal means 207a, 
color interpolation means 207b and image quality correction 
10 means 207c, and the JPEG decoding means 205 transmits JPEG 
decoded image data to the removal means 207a. 

When the retrial module 207 is started, the removal 
means 207a removes part or all of the image quality 
improvement processing, including an interpolation process 
15 (nearest neighbor interpolation, as a specific example) and 
an edge enhancement process, that has been performed for 
the decoded data. 

Thereafter, the data obtained in the removal process 
are transmitted to the color interpolation means 207b, 
20 which then performs another interpolation process and 

another edge enhancement process (fixed edge enhancement 
process with five taps for one line) , and transmits the 
* obtained data to display means 213. 

According to this mode, an example wherein the linear 
25 approximation method or the cubic convolution method is 
used can be employed as another interpolation method. 
Further, an example wherein a flexible edge enhancement 
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filter is used, for local characteristics of an image 
wherein five taps are provided for three lines, can also be 
employed. 

According to this mode, when a retrial of the 
5 interpolation process is required, the retrial module 207 
is started, and when a retrial is not required, the JPEG 
encoding means 205 transmits data, unchanged, to the 
display means 213. The retrial module 207 may be started 
when a retrial of the edge enhancement process is required, 
10 and may perform the interpolation process and the edge 

enhancement process and then perform the processing in the 
above described manner. 

(3) Processing performed by retrial module 
Fig. 4 is a flowchart for explaining the processing 
15 performed by the retrial module. 

When the retrial module is started (S401) , first, the 
module removes part or all of the image quality improvement 
processing, including the interpolation process and the 
edge enhancement process, that have been performed for 
20 input data (S403) . 

Generally, the sequence for the performance of these 
processes consists of a plurality of steps, and whether all 
the processes performed at all the steps should be removed 
or only a process performed at a predetermined step should 
25 be removed can be determined in accordance with the 
situation. 

The processing performed for the input data includes a 
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substantially reversible process and a substantially 
irreversible process. Whether a substantially irreversible 
process (e.g., the color elements of pixels for an image 
that is obtained by performing the interpolation process 
5 for Bayer-arranged data using the nearest neighbor method) 
is performed depends on whether a substantially reversible 
process, i.e., for the values of the original individual 
pixels that were present prior to the interpolation process, 
was performed for the Bayer-arranged data. 

10 The pixels obtained by interpolation and the pixels 

used for the interpolation process, i.e., the color 
elements of the pixels that correspond to the Bayer 
arrangement, are classified, and information relating to 
the original Bayer-arranged data can be recreated based on 

15 the image data. As a result, the correction process can be 
easily removed. 

Actually, there are several variations of the Bayer 
arrangement. Therefore, in order to identify which color 
elements of which pixels were pixels that correspond to the 

20 Bayer arrangement, or which pixels were obtained by 

interpolation,- the pattern of the Bayer arrangement for the 
image sensor of the camera must be stored in the 
information terminal, and must be referred to when the 
retrial process is to be performed. 

25 The same function can be provided in such a manner 

that the camera module outputs the pattern for a Bayer 
arrangement as a camera processing parameter that is 
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accompanied by image data, and a data parser is provided 
for the information terminal so as to separate image data 
from the parameter information and extract the pattern for 
the Bayer arrangement. 
5 Further, to preferably perform the same processing for 

image data that are obtained and transmitted by another 
information terminal, or an image that was previously 
photographed and has been stored, a file format and an 
additional information file are prepared for the storage, 

10 with the image data, of the pattern for a Bayer arrangement. 

It should be noted, however, that the image quality 
can be improved by presuming a default Bayer arrangement, 
instead of obtaining a correct Bayer arrangement that was 
employed, and by preforming the retrial process as 

15 described above. 

When the correction/removal process (S403) is 
completed, a newly employed method for a correction process 
is selected (S405) . The method selection may be manually 
performed by a user, or may be automatically performed by 

20 the terminal in accordance with the use state and various 
conditions . 

In a case wherein, for example, an interpolation 
process, such as a process performed using the nearest 
neighbor method, that does not require a large amount of 
25 operational processing and a great amount of processing 
memory has been performed for the original image data, a 
process by which a natural image is obtained, e.g., a 
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process using the linear approximation method or the cubic 
convolution method, is performed, for example, to improve 
the image quality. In this manner, the re-interpolation 
process can be performed (S409 or S411) . On the other hand, 
5 when the processing speed has a higher priority than has 
the image quality, the re-interpolation process can be 
performed by using the nearest neighbor method (S407) . 
Further, when edge enhancement is required, a corresponding 
process is performed (S415) and the processing is 
10 terminated. 

(4) Explanation for interpolation method employed in 
this mode 

The nearest neighbor method, the linear approximation 
method and the cubic convolution method will now be 

15 described while referring to Figs. 6 to 8 . As is shown in 
Fig. 6, the nearest neighbor method is a method whereby, 
when pixel P0 is to be generated between pixels P2 and P3 
of image data consisting of pixels PI, P2, P3 and P4, the 
pixel value of the nearest pixel P2 is copied to the pixel 

20 value of the pixel P0 (see Fig. 6(b)). Therefore, the 

amount of operational processing is zero, and merely one 
line is sufficient for the processing line memory required 
for the interpolation process. 

On the other hand, as is shown in Fig. 8, the linear 

25 approximation method is a method performed while assuming 

that the pixel values are linearly changed, and whereby the 
pixel value of interpolation target P0 is calculated, by 
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proportionally distributing the pixel values of preceding 
and succeeding pixels P2 and P3 in accordance with the 
distance between the two. That is, when the pixel pitch is 
normalized and the distance to the preceding pixel P2 is 
5 defined as X, the pixel value of the pixel PO is 

represented by weighted addition using the neighboring 
pixels P2 and P3, which is shown in a formula below. In 
this explanation, the process has been performed in a one- 
dimensional direction; however, in actuality, the process 

10 performed for the image is performed horizontally and 
vertically. Since the pixels corresponding to the two 
pixels P2 and P3 are required in both the horizontal 
direction and the vertical direction to obtain the pixel PO, 
a processing memory consisting of a minimum of two image 

15 lines is required. In the following formulas, the pixel 
values are denoted by symbols for corresponding pixels to 
make them easy to understand. 
Ap = X(P3 - P2) 
PO = Ap + P2 

20 = X(P3 - P2) + P2 

= (1 - X) P2 + XP3 .... (1) 

Whereas, according to the cubic convolution method, 
the changes in the pixel values are processed to obtain an 
approximate cubic curve. That is, as is shown in Fig. 7, 

25 according to the cubic convolution method, the pixel value 
of pixel PO is represented by weighted addition using the 
four neighboring pixels PI to P4, as is shown in the 
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formula below. For this explanation, the process has been 
performed in a one-dimensional direction; however, in 
actuality, the process performed for an image is performed 
horizontally and vertically. Since pixels corresponding to 
5 the four pixels PI, P2, P3 and P4 are required both 

horizontally and vertically to obtain the pixel P0, a 
processing memory consisting of a minimum of four image 
lines is required. 

P0 = kl-Pl + k2-P2 + k3-P3 + k4-P4 .... (2) 

10 In this formula, kl to k4 denote weighted coefficients, 

and are obtained by the following interpolation function 
h(t). In this formula, t denotes a variable tl to t4 that 
represents the positional relationship between each of the 
pixels PI to P4 and the processing target pixel P0, and is 
15 represented by using the above described distance X. 

(a + 2 ) 1 1 | 3 - (a + 3)|t| 2 + 10 < |t| < 1 
h(t) = a|t| 3 - 5a|t| 2 + 8a|t| - 4a 1 < |t| < 2 

0 2 < |t| 

.... (3) 

20 wherein tl = 1.0 + X 
t2 = +X 
t3 = 1.0 - X 
t4 = 2.0 - X 

Since the nearest neighbor method is a process merely 
25 for determining a distance from the preceding pixel to the 
succeeding pixel and for setting a pixel value, a drawback 
exists in that the image quality is degraded while the 
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processing speed is extremely short. 

On the other hand, according to formula (1), the 
linear approximation method requires two multiplications 
and one addition, so that compared with the nearest 
5 neighbor method, the amount of operational processing is 
increased, the processing speed is reduced, and a minimum 
two lines of processing line memory and a greater amount of 
memory are required. However, as a characteristic, the 
image quality is superior to that obtained using the 
10 nearest neighbor method. 

Furthermore, the cubic convolution method requires 
four multiplications and three additions even for only the 
weighted addition, so that of these three methods, this 
method requires the largest number of operations, and 
15 accordingly, the processing speed is the lowest. However, 
as a characteristic, the highest image quality is provided. 

Therefore, in the method selection process performed 
by the retrial module (S405 in Fig. 4), the optimal method 
need only be selected from among the above described 
20 methods, so that an image quality and a processing speed 
can be provided in consonance with the use state and the 
purpose of the use. 
2 . Second Mode 

A second mode of the present invention will now be 
25 described while referring to the accompanying drawings. 

Fig. 9 is a diagram showing the external appearance of 
a cellular phone equipped with a camera, for which the 
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present invention is applied- A display panel 2, a ten-key 
pad 3, a function button 4, an antenna 5, etc., are 
provided on the obverse surface of a camera-equipped 
cellular phone 1; a camera unit 11, a battery cover 9, etc., 
are provided on the reverse surface; and a case 6, which 
holds all these components, is provided. Further, a data 
output terminal 41 is provided for the connection of an 
external printer. As is well known, a camera-equipped 
cellular phone 1 is small and light, so that no 
inconvenience is incurred when the cellular phone is 
carried in a handbag. The camera unit 11 includes a lens 7 
and an LED luminaire 8, and is constituted as a camera 
module having an independent case 10. The camera unit is 
provided as an independent module because multiple uses for 
the camera module are available since the camera module can 
be easily assembled with another cellular phone or a PDA. 
Therefore, the camera-equipped cellular phone 1 of this 
mode can be separated into the camera module 11 and the 
remaining section (the host module) . 

The function button 4, for example, is used to send or 
receive a call, or is used as a shutter button for 
photographing. The function button 4 is also used as a 
button to start the reconstruction of image data. When a 
user employs the camera-equipped cellular phone 1 to make a 
call, the user holds the camera-equipped cellular phone 1 
with the lens 7 aimed at a target, and confirms, on the 
display panel 2, a preview image obtained by the camera 
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module 11. Then, imaging is performed by pressing the 
function button 4, and the obtained image data are stored 
in the storage device provided for the camera-equipped 
cellular phone 1 . 
5 While referring to Fig. 10, an explanation will be 

given for the hardware configuration and the operation of 
the camera-equipped cellular phone for which the present 
invention is applied. As is described above, the camera- 
equipped cellular phone 1 is constituted by the camera 

10 module 11 and a host module 12; the camera module 11 

controls the imaging and the creation of image data, and 
the host module 12 controls the saving and the display of 
the created image data and PDA functions, such as a phone 
function and a scheduling function. 

15 The camera module 11 includes the lens 7, the LED 

luminaire 8, a solid state imaging device 13, an A/D 
converter 14, an image data construction unit 16, a JPEG 
compression unit 17, a bus 18, a data interface 19 and a 
control interface 20. Of these components, the image data 

20 construction unit 16, the JPEG compression unit 17, the bus 
18, the data interface 19 and the control interface 20 are 
provided as a camera LSI 15 that uses a single chip. 

A CCD or a CMOS sensor, for example, can be employed 
as the solid state imaging device) . The solid-state 

25 imaging device 13 performs imaging by converting, into an 

electrical signal, light that has passed through the lens 7. 
The signal output by the solid state imaging device 13 is 
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converted into digital data by the A/D converter 14. The 
digital data are called Bayer-type data, and are not yet 
image data that can be displayed by a computer or that can 
be printed by a printer. 
5 Image data are constructed by the image data 

construction unit 16. The image data construction unit 16 
first performs the original image processing, such as lens 
density correction and white balance control, for the 
Bayer-type data. Then, the Bayer-type data for which the 

10 original image processing has been performed are separated 
into red elements (R) , green elements (G) and blue elements 
(B) , and the CFA (Color Filter Array) interpolation process 
is performed for these color elements to construct RGB 
image data consisting of three RGB planes. 

15 The algorithm used for the CFA interpolation very much 

affects the quality of constructed image data. It is 
difficult for image data having a high quality to be 
obtained using an algorithm available for a camera LSI that 
limits size, cost and speed. The image data construction 

20 unit 16 performs the processing, such as edge enhancement 
or gamma correction, for the thus obtained image data. 

Finally, the RGB form of the image data is changed to 
the YUV form. The image data constitutina one frame are 
sequentially produced for each line or for every several 

25 lines, and in the end, the image data for the whole frame 

are constructed by employing signals that are output by the 
solid state imaging device 13 using single image 



- 25 - 



photography. Each set of the thus produced image data is 
sequentially transmitted to the JPEG compression unit 17 , 
which then performs JPEG compression for the image data and 
transmits the resultant data, through the data interface 19, 
5 to the host module 12. 

Referring again to Fig. 10, the hardware configuration 
and the operation of the host module 12 will be described. 
The host module 12 includes a data interface 25, a control 
interface 26, a CPU 30, a bus 24, an interface 31 for a 

10 temporary storage device, a temporary storage device 32, an 
interface 33 for a main storage device, a main storage 
device 34 , an interface 35 for a display device, a display 
device 36, an interface 37 for a keyboard, a keyboard 38, 
an interface 40 for a printer, a baseband controller 22, 

15 and an antenna 23. 

Of these components, the CPU 30, the bus 24 and the 
interfaces 25, 26, 31, 33, 35, 37 and 40 are provided as an 
application engine 21 by using a single chip. The baseband 
controller 22 controls a function related to the sending or 

20 the reception of calls, and has a special CPU. The 

application engine 21 controls a function other than the 
call transmission or reception function, and also controls 
functions for the processing of image data received from 
the camera module 11, for controlling the keyboard 38, for 

25 playing games, for playing music, and for scheduling. The 
CPU 30 can provide, through the control interfaces 26 and 
20 and the bus 18, not only control of the JPEG compression 
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unit 17 of the camera module 11, but also control of the 
individual processes performed by the image data 
construction unit 16, the turning on or off of the LED 
luminaire 8, the changing of the data collection mode of 
5 the solid state imaging device, and parameter control for 
the A/D converter 14. The keyboard 38 includes a ten-key 
pad 3 and a function button 4. 

The image data output by the camera module 11 are 
transmitted through the data interface 2b to the host 

10 module 12, and are temporarily stored in the temporary 

storage device 32. A typical temporary storage device 32 
is an SDRAM. The CPU 30 thereafter reads image data from 
the temporary storage device 32, and stores the image data 
in the main storage device 34 . The main storage device 34 

15 includes a storage medium on which data can be continuously 
saved when the camera-equipped cellular phone 1 has been 
powered off, and can, for example, be a flash memory, a CF 
card or an SD card. 

The camera-equipped cellular phone of the present 

20 invention can reconstruct image data by using the host 

module. The CPU 30 performs the reconstruction of image 
data upon the reception of an instruction from a user 
interface. The CPU 30 first processes the image data to 
restore the Bayer-type data, and then employs a high-level 

25 algorithm to construct new image data. The thus 

constructed high quality image data are then stored in the 
main storage device 34, or are transmitted to a printer 
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through the printer interface 40. 

The processing for the construction and reconstruction 
of image data will be described while referring to Fig. 11. 
Data obtained merely by digitizing the output signal 
5 of the sensor, as is indicated by number 1144, is of Bayer 
type, wherein multiple unit matrixes, each of which is 
composed of one red pixel (R) , two green pixels (G) and one 
glue pixel (B) , are repeated. The elements of the 
individual colors are separated from the Bayer-type data, 

10 and an interpolation process is performed for these 
elements to construct RGB image data 45. Pixel data 
enclosed by circles, as denoted by number 1145, are either 
the same as the original Bayer-type data or are 
substantially unchanged. Therefore, the pixel data 

15 enclosed by circles, denoted by 1145, need only be 

extracted to reconstruct the original Bayer-type data. 

Of course, in order to reconstruct the original Bayer- 
type data, which pixel data of the RGB image data are 
derived from the original Bayer-type data must be 

20 determined in advance. There are various methods available 
for this determination: a method for performing this 
determination at the interpolation stage; and a method for 
embedding information in the header of the constructed 
image data. When the high-level original image processing, 

25 the interpolation process and the post-process are 

performed for the reconstructed Bayer-type data, new image 
data 1147 are created. 
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It should be noted that when the Bayer-type data are 
reconstructed from the JPEG compressed RGB data, Bayer-type 
data that are completely the same as the original data can 
not be extracted. However, the advantage obtained when the 
5 image data are reconstructed by using a high-level 
algorithm is greater than the loss of some data. 

The processing performed by the camera-equipped 
cellular phone 1 of the present invention will be now 
described while referring to Fig. 12. 

10 In the case of photography, first, the user interface 

is manipulated (S1201) . Then, the image sensor of the 
camera module 11 converts light into an electrical signal 
(S1202), and image data are constructed based on the 
electrical signal (S1203) . The thus constructed image data 

15 are compressed using the JPEG method (S1204), and the 
compressed data are transmitted to the host module 12 
(S1205) . The host module receives the image data (S1206) , 
and stores the image data in the storage device (S1207). 
In the case of the printing of obtained image data, 

20 first, the user interface is manipulated (S1208) . Then, 

the host module 12 reads image data from the storage device 

(51209) , decompresses the imaqe data that have been read 

(51210) , and extracts and reconstructs Bayer-type data 
based on the decompressed image data (S1211). Further, the 

\ 25 host module 12 employs a new high-level algorithm to 

prepare new image data based on the thus reconstructed 
Bayer-type data (step S1212) , and outputs the newly created 
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image data to the printer (step S1213) . 

The modes of the present invention have been 
explained; however, the present invention is not limited to 
the above description. While the nearest neighbor method, 
5 the linear approximation method or the cubic convolution 
method has been employed in these modes as the 
interpolation method, an interpolation process using, for 
example, the spline function or the Bezier function can be 
employed instead of, or in addition to the above methods. 

10 Furthermore, in the modes, the present invention has 

been applied for the image data processing. However, the 
present invention is not limited to this, and can be 
applied for a case wherein there is a change in the 
sampling pitch of various data, e.g., the sampling 

15 frequency of audio data. 

Industrial Applicability 
As is described above, according to the present 
invention, an information terminal can be provided that can 
change, at a low cost and within a short period of time, 

20 image quality and processing speed in accordance with the 
use state and a variety of conditions. 

Furthermore, according to the present invention, an 
information terminal can be provided that can reduce the 
affect produced by the simplified processing performed by 

25 the camera DSP, and that can improve the image quality of 
an image. 

Further, according to the present invention, . an. 
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information terminal can be provided that can produce an 
image having a higher image quality than other information 
terminals that employ cameras having the same functions. 

In addition, according to the present invention, an 
information terminal can be provided that can improve the 
quality of an image without changing a camera module or 
increasing the functions required of the camera DSP. 



